Papers with ranking metric
Permutative Preference Alignment from Listwise Ranking of Human Judgments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to align Large Language Models with human preferences are based on the Bradley-Terry model, but when multiple responses are available, the B-T model fails to guarantee an accurate list ranking of the responses. |
| Approach: | They propose an offline listwise approach that incorporates the Normalized Discounted Cumulative Gain (NDCG) as an alternative training objective for LLM alignment. |
| Outcome: | The proposed approach outperforms existing pairwise and listwise methods on evaluation sets and general benchmarks such as AlpacaEval. |
A Study of Latent Structured Prediction Approaches to Passage Reranking (N19-1)
Copied to clipboard
| Challenge: | a structured output framework is useful for learning to rank problems . current approaches for answer sentence reranking are mostly based on pairwise ranking signals or simple binary classification. |
| Approach: | They propose a structured output approach which regards rankings as latent variables . they propose an inference procedure to find the max-violating ranking based on decomposition of the corresponding loss. |
| Outcome: | The proposed approach solves the optimization problem on WikiQA and TREC13 datasets. |
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been experiencing seismic growth in size and capabilities, radically transforming the field of NLP. |
| Approach: | They propose a generalized variant of iterative self-critique and self-refinement devoid of external influence and a ranking metric to find the optimal model for a given task considering refined performance and cost. |
| Outcome: | The proposed model improves 8.2% from baseline and even with extremely small memory footprints, outperforms ChatGPT post-refinement. |